Interpretation of Multimodal Designation with Imprecise Gesture
نویسندگان
چکیده
We are interested in multimodal systems that use the following modes and modalities: speech (and natural language) as input as well as output, gesture as input and visual as output using screen displays. The user exchanges with the system by gesture and/or oral statements in natural language. This exchange, encoded in the different modalities, carries the goal of the user and also the designation of objects (referents) needed to achieve this goal. The system must identify in a precise and non-ambiguous way the objects designated by the user. In this paper, our main concern is the multimodal designations, with possibly imprecise gesture, of objects in the visual context. In order to identify such a designation, we propose a solution which uses probabilities, knowledge about manipulated objects, and perceptive aspects (degree of salience) associated with these objects.
منابع مشابه
Co-Production of Speech and Pointing Gestures in Clear and Perturbed Interactive Tasks: Multimodal Designation Strategies
Designation consists in attracting an interlocutor’s attention on a specific object and/or location. It is most often achieved using both speech (e.g., demonstratives) and gestures (e.g., manual pointing). This study aims at analyzing how speech and pointing gestures are co-produced in a semi-directed interactive task involving designation. 20 native speakers of French were involved in a cooper...
متن کاملClues for the Identification of Implicit Information in Multimodal Referring Actions – DRAFT VERSION
The implicit is an imprecise and heterogeneous notion that plays a role, not only in the global comprehension of utterances, but also in the interpretation of reduced phenomena like multimodal referring actions, which combine visual perception, language, and gesture. The identification of the intended referents relies on the correct identification of implicit information that is communicated wi...
متن کاملMind: a Context-based Multimodal Interpretation Framework in Conversational Systems
In a multimodal human-machine conversation, user inputs are often abbreviated or imprecise. Simply fusing multimodal inputs together may not be sufficient to derive a complete understanding of the inputs. Aiming to handle a wide variety of multimodal inputs, we are building a context-based multimodal interpretation framework called MIND (Multimodal Interpreter for Natural Dialog). MIND is uniqu...
متن کاملDiscourse Coherence and Gesture Interpretation
In face-to-face conversation, communicators orchestrate multimodal contributions that meaningfully combine the linguistic resources of spoken language and the visuo-spatial affordances of gesture. In this paper, we characterise this meaningful combination in terms of the COHERENCE of gesture and speech. Descriptive analyses illustrate the diverse ways gesture interpretation can supplement and e...
متن کاملFrom a Wizard of Oz experiment to a real time speech and gesture multimodal interface
This paper describes a Wizard of Oz cooperative story telling experiment named Virstory, where user speech-gesture actions are interpreted in order to cooperatively build a story with another person, partner of the interpreter. The gesture, speech and multimodal behaviours of 20 subjects are detailed. The Multimodal Oral With Gesture Large display Interface (MOWGLI) is then described. It is an ...
متن کامل